Skip to main content
SpringerLink
Log in

Dual Adversarial Network: Toward Real-World Noise Removal and Noise Generation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12355))

Included in the following conference series:

Abstract

Real-world image noise removal is a long-standing yet very challenging task in computer vision. The success of deep neural network in denoising stimulates the research of noise generation, aiming at synthesizing more clean-noisy image pairs to facilitate the training of deep denoisers. In this work, we propose a novel unified framework to simultaneously deal with the noise removal and noise generation tasks. Instead of only inferring the posteriori distribution of the latent clean image conditioned on the observed noisy image in traditional MAP framework, our proposed method learns the joint distribution of the clean-noisy image pairs. Specifically, we approximate the joint distribution with two different factorized forms, which can be formulated as a denoiser mapping the noisy image to the clean one and a generator mapping the clean image to the noisy one. The learned joint distribution implicitly contains all the information between the noisy and clean images, avoiding the necessity of manually designing the image priors and noise assumptions as traditional. Besides, the performance of our denoiser can be further improved by augmenting the original training dataset with the learned generator. Moreover, we propose two metrics to assess the quality of the generated noisy image, for which, to the best of our knowledge, such metrics are firstly proposed along this research line. Extensive experiments have been conducted to demonstrate the superiority of our method over the state-of-the-arts both in the real noise removal and generation tasks. The training and testing code is available at https://github.com/zsyOAOA/DANet.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The phrase “noise generation” indicates the generation process of noisy image from clean image throughout this paper.

  2. 2.

    We mildly assume that \(\textit{\textbf{y}} \sim p(\textit{\textbf{y}})\) is easily implemented by sampling \(\textit{\textbf{y}}\) from the empirical distribution \(p(\textit{\textbf{y}})\) of the training data set, and so does as \(\textit{\textbf{x}} \sim p(\textit{\textbf{x}})\).

  3. 3.

    https://www.eecs.yorku.ca/~kamel/sidd/benchmark.php.

  4. 4.

    https://noise.visinf.tu-darmstadt.de/benchmark/.

References

  1. Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  2. Agostinelli, F., Anderson, M.R., Lee, H.: Adaptive multi-column deep neural networks with application to robust image denoising. Adv. Neural Inf. Process. Syst. 26, 1493–1501 (2013)

    Google Scholar 

  3. Anaya, J., Barbu, A.: Renoir - a benchmark dataset for real noise reduction evaluation. arXiv preprint arXiv:1409.8230 (2014). https://academic.microsoft.com/paper/1514812871

  4. Anwar, S., Barnes, N.: Real image denoising with feature attention. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3155–3164 (2019)

    Google Scholar 

  5. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)

  6. Barbu, A.: Training an active random field for real-time image denoising. IEEE Trans. Image Process. 18(11), 2451–2462 (2009)

    Article  MathSciNet  Google Scholar 

  7. Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11036–11045 (2019)

    Google Scholar 

  8. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 60–65. IEEE (2005)

    Google Scholar 

  9. Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2392–2399 (2012)

    Google Scholar 

  10. Cao, X., et al.: Low-rank matrix factorization under general mixture noise distributions. In: The IEEE International Conference on Computer Vision (ICCV), December 2015

    Google Scholar 

  11. Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3155–3164 (2018)

    Google Scholar 

  12. Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2017)

    Article  Google Scholar 

  13. Chongxuan, L., Xu, T., Zhu, J., Zhang, B.: Triple generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 4088–4098 (2017)

    Google Scholar 

  14. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  15. Dong, W., Shi, G., Li, X.: Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22(2), 700–711 (2012)

    Article  MathSciNet  Google Scholar 

  16. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  17. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869 (2014)

    Google Scholar 

  18. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein gans. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)

    Google Scholar 

  19. Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1712–1722 (2019)

    Google Scholar 

  20. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  21. Huiskes, M.J., Thomee, B., Lew, M.S.: New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative. In: Proceedings of the International Conference on Multimedia Information Retrieval, pp. 527–536 (2010)

    Google Scholar 

  22. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  23. Jain, V., Seung, S.: Natural image denoising with convolutional networks. Adv. Neural Inf. Process. Syst. 21, 769–776 (2008)

    Google Scholar 

  24. Jaroensri, R., Biscarrat, C., Aittala, M., Durand, F.: Generating training data for denoising real rgb images via camera pipeline simulation. arXiv preprint arXiv:1904.08825 (2019)

  25. Kim, D.W., Ryun Chung, J., Jung, S.W.: GRDN: grouped residual dense network for real image denoising and gan-based real-world noise modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0 (2019)

    Google Scholar 

  26. Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: ICLR 2015: International Conference on Learning Representations 2015 (2015). https://academic.microsoft.com/paper/2964121744

  27. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR 2014: International Conference on Learning Representations (ICLR) 2014 (2014)

    Google Scholar 

  28. Lebrun, M., Buades, A., Morel, J.M.: A nonlocal Bayesian image denoising algorithm. SIAM J. Imaging Sci. 6(3), 1665–1688 (2013)

    Article  MathSciNet  Google Scholar 

  29. Liu, D., Wen, B., Fan, Y., Loy, C.C., Huang, T.: Non-local recurrent network for image restoration. In: NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems, pp. 1673–1682 (2018)

    Google Scholar 

  30. Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 2272–2279 (2009)

    Google Scholar 

  31. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2007)

    Article  MathSciNet  Google Scholar 

  32. Mao, X.J., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: NIPS 2016 Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 2810–2818 (2016)

    Google Scholar 

  33. Meng, D., De La Torre, F.: Robust matrix factorization with unknown noise. In: The IEEE International Conference on Computer Vision (ICCV), December 2013

    Google Scholar 

  34. Nam, S., Hwang, Y., Matsushita, Y., Joo Kim, S.: A holistic approach to cross-channel image noise modeling and its application to image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1683–1691 (2016)

    Google Scholar 

  35. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)

    Google Scholar 

  36. Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1586–1595 (2017)

    Google Scholar 

  37. Pıtz, T., Roth, S.: Neural nearest neighbors networks. In: NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems, pp. 1087–1098 (2018)

    Google Scholar 

  38. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR 2016: International Conference on Learning Representations 2016 (2016). https://academic.microsoft.com/paper/2963684088

  39. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015). https://academic.microsoft.com/paper/1901129140

  40. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1–4), 259–268 (1992)

    Article  MathSciNet  Google Scholar 

  41. Samuel, K.G.G., Tappen, M.F.: Learning optimized map estimates in continuously-valued MRF models. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 477–484 (2009)

    Google Scholar 

  42. Schmidt, U.: Half-quadratic inference and learning for natural images (2017)

    Google Scholar 

  43. Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR 2014 Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774–2781 (2014)

    Google Scholar 

  44. Sun, J., Tappen, M.F.: Learning non-local range Markov random field for image restoration. In: CVPR 2011, pp. 2745–2752 (2011)

    Google Scholar 

  45. Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557 (2017)

    Google Scholar 

  46. Tsin, Y., Ramesh, V., Kanade, T.: Statistical calibration of CCD imaging process. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 1, pp. 480–487. IEEE (2001)

    Google Scholar 

  47. Wang, R., Chen, B., Meng, D., Wang, L.: Weakly supervised lesion detection from fundus images. IEEE Trans. Med. Imaging 38(6), 1501–1512 (2018)

    Article  Google Scholar 

  48. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  49. Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. Adv. Neural Inf. Process. Syst. 25, 341–349 (2012)

    Google Scholar 

  50. Xie, Q., Zhao, Q., Meng, D., Xu, Z.: Kronecker-basis-representation based tensor sparsity and its applications to tensor recovery. IEEE Trans. Pattern Anal. Mach. Intell. 40(8), 1888–1902 (2017)

    Article  Google Scholar 

  51. Xu, J., Li, H., Liang, Z., Zhang, D., Zhang, L.: Real-world noisy image denoising: a new benchmark. arXiv preprint arXiv:1804.02603 (2018)

  52. Xu, J., Zhang, L., Zhang, D.: A trilateral weighted sparse coding scheme for real-world image denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 21–38. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_2

    Chapter  Google Scholar 

  53. Xu, J., Zhang, L., Zhang, D., Feng, X.: Multi-channel weighted nuclear norm minimization for real color image denoising. In: ICCV (2017)

    Google Scholar 

  54. Yong, H., Meng, D., Zuo, W., Zhang, L.: Robust online matrix factorization for dynamic background subtraction. IEEE Trans. Pattern Anal. Mach. Intell. 40(7), 1726–1740 (2017)

    Article  Google Scholar 

  55. Yue, Z., Yong, H., Meng, D., Zhao, Q., Leung, Y., Zhang, L.: Robust multiview subspace learning with nonindependently and nonidentically distributed complex noise. IEEE Trans. Neural Netw. Learn. Syst. 31(4), 1070–1083 (2019)

    Google Scholar 

  56. Yue, Z., Yong, H., Zhao, Q., Meng, D., Zhang, L.: Variational denoising network: toward blind noise modeling and removal. In: Advances in Neural Information Processing Systems, pp. 1688–1699 (2019)

    Google Scholar 

  57. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  Google Scholar 

  58. Zhou, M., Chen, H., Ren, L., Sapiro, G., Carin, L., Paisley, J.W.: Non-parametric Bayesian dictionary learning for sparse image representations. In: Advances in Neural Information Processing Systems, pp. 2295–2303 (2009)

    Google Scholar 

  59. Zhu, F., Chen, G., Hao, J., Heng, P.A.: Blind image denoising via dependent Dirichlet process tree. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1518–1531 (2016)

    Article  Google Scholar 

  60. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Xi’an Jiaotong University, Shaanxi, China

    Zongsheng Yue, Qian Zhao & Deyu Meng

  2. Hong Kong Polytechnic University, Hong Kong, China

    Zongsheng Yue & Lei Zhang

  3. DAMO Academy, Alibaba Group, Shenzhen, China

    Lei Zhang

  4. The Macau University of Science and Technology, Macau, China

    Deyu Meng

Authors
  1. Zongsheng Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qian Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deyu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deyu Meng .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1367 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yue, Z., Zhao, Q., Zhang, L., Meng, D. (2020). Dual Adversarial Network: Toward Real-World Noise Removal and Noise Generation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12355. Springer, Cham. https://doi.org/10.1007/978-3-030-58607-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58607-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58606-5

  • Online ISBN: 978-3-030-58607-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation